Skip to content

Add External Competitions feature - #2508

Merged
Didayolo merged 8 commits into
developfrom
external_competitions
Sep 17, 2026
Merged

Didayolo merged 8 commits into
developfrom
external_competitions

Conversation

@ihsaan-ullah

@ihsaan-ullah ihsaan-ullah commented Aug 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • This PR adds a new External Competitions feature: Codabench can periodically fetch and list public competitions from other registered Codabench/CodaLab instances.
  • New external_competitions app with ExternalPlatform, ExternalCompetition, ExternalFetchLog models, separate fetchers for codalab and codabench instances, and a daily Celery sync task that fetches competitions from all active platforms. Detailed logs are recorded for each platform.
  • Fetching external competitions could result in success, failure, or partial-success. Partial success is useful in cases where we receive some pages of competitions but fails later on. In this case we save what we have and skip deletion sync because of partial success.
  • New API endpoints (/api/external_competitions/, /api/external_competitions/platforms/) with search and platform filtering.
  • New public page + banner on the existing public competitions/benchmarks page, linking users to the External Competitions view.
  • A banner on eternal competitions page points users to docs for registering their platforms with codabench.
  • A flag in .env turns this feature on/off. EXTERNAL_COMPETITIONS_ENABLED (off by default). If turned off, routes, pages, apis, and fetching does not work.
  • Detailed tests are added for the apis, fetchers and the whole feature.
  • New docs page covering setup for self-hosters and a registration/unregistration process for other platform maintainers, linked from the deploy guide.

Screenshots

External competition banner in public benchmarks page
Screenshot 2026-08-22 at 4 03 04 PM

External competitions page
Screenshot 2026-08-27 at 3 02 44 PM

Run fetch

to run the fetch manually, use the following code

docker compose exec django ./manage.py shell -c "from external_competitions.tasks import fetch_external_competitions; fetch_external_competitions()"

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@ihsaan-ullah

Copy link
Copy Markdown
Collaborator Author

External page now is a bit different from public competitions page. For a comparison see screenshots below:

Public comp page
Screenshot 2026-09-03 at 10 38 19 PM

External comp page
Screenshot 2026-09-03 at 10 36 52 PM

@Didayolo
Didayolo self-requested a review September 8, 2026 13:06
@Didayolo Didayolo self-assigned this Sep 8, 2026
@Didayolo
Didayolo force-pushed the external_competitions branch from bf60f59 to c2601db Compare September 10, 2026 02:15
ihsaan-ullah and others added 7 commits September 17, 2026 13:01
…Codabench/CodaLab instances

New app — external_competitions
- Models: ExternalPlatform (a registered remote instance), ExternalCompetition (a fetched competition), ExternalFetchLog (per-sync audit trail with SUCCESS/PARTIAL_SUCCESS/FAILURE status, counts, and error messages)
- admin.py registrations for all three models with list filters/search
- Initial migration

Fetchers
- fetchers/codabench_fetcher.py — paginated fetch from a Codabench instance's public competitions API, with a page-fetch throttle and a MAX_PAGES safety cap
- fetchers/codalab_fetcher.py — flat-list fetch from a CodaLab instance
- fetchers/exceptions.py — PartialFetchError, letting a fetcher hand back whatever it collected before a later page failed

Sync task
- fetch_sync.py — fetch_external_competitions Celery task (feature-flag gated, scheduled daily via CELERY_BEAT_SCHEDULE) and sync_platform, which diffs fetched data against the DB (create/update/delete), skipping the delete step on a partial fetch to avoid dropping valid competitions from unfetched pages

API
- api/serializers/external_competitions.py — ExternalCompetitionSerializer, ExternalPlatformFilterSerializer
- api/views/external_competitions.py — list view with search/platform filtering, and an unpaginated platforms list for the filter UI
- api/urls.py — both routes registered only when EXTERNAL_COMPETITIONS_ENABLED is on

Frontend
- external_competitions/urls.py + views.py + templates/external_competitions/public.html — the public page route
- static/riot/external_competitions/external_competition_list.tag — the competition list/filter/pagination UI
- client.js — API helpers for the two new endpoints
- public-list.tag — banner linking to the External Competitions page (shown only when the feature is enabled)
- base.html / context_processors.py — expose the feature flag and public URL to the frontend

Settings
- EXTERNAL_COMPETITIONS_ENABLED flag (settings/base.py, .env_sample), off by default
- external_competitions added to INSTALLED_APPS

Test data
- factories.py — ExternalPlatformFactory, ExternalCompetitionFactory

Tests
- external_competitions/tests/test_fetchers.py — unit tests for both fetchers (pagination, throttling, MAX_PAGES, error propagation, partial-fetch handling)
- external_competitions/tests/test_fetch_sync.py — unit tests for sync_platform/fetch_external_competitions (create/update/delete diffing, partial-success handling, per-platform failure isolation)
- api/tests/test_external_competitions.py — API tests for both endpoints, plus flag-gated URL registration tests

Documentation
- New External-Competitions.md page: enabling the flag, admin fields, sync mechanics, and registration/unregistration instructions for other platform maintainers
- zensical.toml nav entry
- Tip added to the deploy guide pointing self-hosters at registering their instance
…iodically, updated codabench fetcher to not use the next url from the response but instead use a page counter for next page, related updates to docs and tests
@Didayolo
Didayolo force-pushed the external_competitions branch from a61b0df to 1f21663 Compare September 17, 2026 11:01
@Didayolo
Didayolo merged commit 22b9e08 into develop Sep 17, 2026
1 check passed
@Didayolo
Didayolo deleted the external_competitions branch September 17, 2026 11:16
Didayolo added a commit that referenced this pull request Sep 18, 2026
* Update version.json

* add option to forbid the compute worker from pulling the competition image automatically

* rename feature name

* packages bump via uv lock --upgrade; bump django 5.2.15 -> 5.2.16 (patch)

* rebase and update some packages via uv lock --upgrade

* various packages upgrades with uv lock --upgrade

* Add pagination to GET /api/competitions and fix participating_in leak

Paginate the competition list endpoint (LargePagination), and update
the Organizing/Participating tabs in competition_list.tag to
consume the new {count, next, previous, results} shape with
next/previous buttons (hidden on a single page). Fix the navbar
competition search to read response.results instead of assuming a
bare array.

Fix participating_in filter incorrectly including competitions the
user organizes: creators/collaborators are auto-added as approved
participants (Competition.save()), so they leaked into the
Participating tab. Applied the same fix to the public competitions
endpoint's participating_in filter and updated its tests. Added
coverage for the list endpoint's pagination and participating_in
behavior. Minor renames for clarity (change_page, tab labels).

* remove ram limit for site_worker

* update limit to 15 GB, same as default quota

* Update docker-compose.yml

* queue creation error improved

* allow users to add collaborators to queues when the queue adding limit is reached. Fixed user lookup to not show deleted users in the collaborators dropdown

* Remove robot submissions and bot user feature

Remove the is_bot user flag and allow_robot_submissions competition setting, along with the permission bypasses they granted (unlimited submissions, auto-approval as participant, and unrestricted re-run/secret-check bypass on any submission). Also removes the related UI, docs, and tests.

* Fix migrations

* privacy link updated

* Fix organization name not showing on leaderboard for multi-task competitions

Propagate organization when creating chiled submissions during multi-task split, phase migration, and re-run so that leaderboard entries for organization submissions on multi-task competitions show the organization name instead of falling back to the submitter's username.

* Merge pull request #2487 from codalab/throttling

API Throttling

* restrict get_details to submission owner or admin

Anonymous and non-owner requests could previously view submission
details when hide_output was off. Now only the owner or an admin
can access them. Adds tests covering the fix.

* Switch to quay.io for mc and minio images (#2537)

* switch to quay.io for mc and minio images

* Update documentation

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>
Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Deny anonymous access to /api/submissions/ list and retrieve (#2536)

* Deny anonymous access to /api/submissions/ list and retrieve

Anonymous GETs returned full submission records (filenames, status
details, fact sheet answers) for any finished leaderboard submission,
ignoring is_public. Now returns an empty queryset instead; public
leaderboard data is already served by get_leaderboard in PhaseViewSet

* Add missing raise

* Fix test and remove debug prints

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Packages upgrade (#2535)

* update djangorestframework in pyproject and packages in uv.lock with uv lock --upgrade

* update boto3 (removed s3transfer and botocore, they are dependencies), update gunicorn and uvicorn to latest

* update packages in pyproject

* update django-oauth-toolkit and related packages

* update dompurify

* add DOMPurify to marked

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>

* Submission API: Enforce phase start/end window on submission creation (#2496)

* Enforce phase start/end window on submission creation

Phase.is_active returned True unconditionally when end was unset,
never checking start — a not-yet-started phase with no end date was
treated as active. SubmissionCreationSerializer.validate() also never
checked is_active or can_user_make_submissions(), so submissions could
be created via the API before a phase started or after it ended.

- Fix Phase.is_active to check start regardless of whether end is set
- Reject submission creation when the target phase is not active or
  the user has hit their submission limit, returning a 400 error
- Add tests for both new fixes

* Fix flake8

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* compute worker workaround for api throttling

* Add External Competitions feature (#2508)

* Add External Competitions feature: aggregate competitions from other Codabench/CodaLab instances

New app — external_competitions
- Models: ExternalPlatform (a registered remote instance), ExternalCompetition (a fetched competition), ExternalFetchLog (per-sync audit trail with SUCCESS/PARTIAL_SUCCESS/FAILURE status, counts, and error messages)
- admin.py registrations for all three models with list filters/search
- Initial migration

Fetchers
- fetchers/codabench_fetcher.py — paginated fetch from a Codabench instance's public competitions API, with a page-fetch throttle and a MAX_PAGES safety cap
- fetchers/codalab_fetcher.py — flat-list fetch from a CodaLab instance
- fetchers/exceptions.py — PartialFetchError, letting a fetcher hand back whatever it collected before a later page failed

Sync task
- fetch_sync.py — fetch_external_competitions Celery task (feature-flag gated, scheduled daily via CELERY_BEAT_SCHEDULE) and sync_platform, which diffs fetched data against the DB (create/update/delete), skipping the delete step on a partial fetch to avoid dropping valid competitions from unfetched pages

API
- api/serializers/external_competitions.py — ExternalCompetitionSerializer, ExternalPlatformFilterSerializer
- api/views/external_competitions.py — list view with search/platform filtering, and an unpaginated platforms list for the filter UI
- api/urls.py — both routes registered only when EXTERNAL_COMPETITIONS_ENABLED is on

Frontend
- external_competitions/urls.py + views.py + templates/external_competitions/public.html — the public page route
- static/riot/external_competitions/external_competition_list.tag — the competition list/filter/pagination UI
- client.js — API helpers for the two new endpoints
- public-list.tag — banner linking to the External Competitions page (shown only when the feature is enabled)
- base.html / context_processors.py — expose the feature flag and public URL to the frontend

Settings
- EXTERNAL_COMPETITIONS_ENABLED flag (settings/base.py, .env_sample), off by default
- external_competitions added to INSTALLED_APPS

Test data
- factories.py — ExternalPlatformFactory, ExternalCompetitionFactory

Tests
- external_competitions/tests/test_fetchers.py — unit tests for both fetchers (pagination, throttling, MAX_PAGES, error propagation, partial-fetch handling)
- external_competitions/tests/test_fetch_sync.py — unit tests for sync_platform/fetch_external_competitions (create/update/delete diffing, partial-success handling, per-platform failure isolation)
- api/tests/test_external_competitions.py — API tests for both endpoints, plus flag-gated URL registration tests

Documentation
- New External-Competitions.md page: enabling the flag, admin fields, sync mechanics, and registration/unregistration instructions for other platform maintainers
- zensical.toml nav entry
- Tip added to the deploy guide pointing self-hosters at registering their instance

* flake-8 fixes

* registration instructions added to external competitions page

* renamed fetch_sync.py to tasks.py so that celery can run the task periodically, updated codabench fetcher to not use the next url from the response but instead use a page counter for next page, related updates to docs and tests

* ui updates to differentiate external page from public page.

* Some small fixes

* Fix spinner

* Avoid duplicated CSS

---------

Co-authored-by: didayolo <adrien.pavao@gmail.com>

* Option to stop compute worker from sending logs and/or prediction files to Codabench (#2460)

* add option in compute worker env to not send logs to the instance, instead writing them in a local file

* rename the No Cleanup env variable, add documentation

* use real boolean values

* update logs_loguru to inclue new tasks variable names to color them

* change variable name to use boolean

* fix some syntax

* add option to forbid the compute worker from sending predictions to the codabench instance storage

* add better coloration for some logs in the compute worker

* update documentation

* add error when copying failes when in no prediction upload mode

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>

* Revert "Compute worker workaround for API throttling"

* Revert "Merge pull request #2487 from codalab/throttling"

This reverts commit 17416df.

---------

Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>
Co-authored-by: wlln <105347618+wlln@users.noreply.github.com>
Co-authored-by: Adrien Pavão <adrien.pavao@gmail.com>
Co-authored-by: Ihsan Ullah <ihsan2131@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants